SCAM Benchmark Results

SCAM (Security Comprehension Agentic Measure) is an open benchmark that tests whether AI agents protect users when they have access to real-world tools like email, credential stores, and web forms.

Unlike static classification benchmarks, every scenario here is a multi-turn conversation where the agent must independently recognize a threat, warn the user, and refuse to carry out dangerous actions. Scenarios cover phishing, social engineering, credential exposure, e-commerce scams, data leakage, and multi-stage attacks.

This evaluation runs each scenario twice per model: once without any guidance (baseline) and once with a security skill prepended to the system prompt (skill). The difference measures how much targeted instructions improve safety.